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ABSTRACT 



This study examined the validity of using the Child Behavior 
Checklist (CBCL) Thought Problem sub-scale with urban low-income children 
(N=46) referred to a hospital -based mental health clinic. It was hypothesized 
that cultural, linguistic, or socio-economic status (SES) factors may 
influence the manner in which parents understand and respond to items on the 
Thought Problem' Scale . Preliminary analysis of CBCL data indicated that more 
than 33% of the children were in the clinical range for thought problems in 
contrast to less than 5% of the same sample who were diagnosed by a clinical 
psychiatrist or therapist. The CBCL items that appear sensitive to 
misinterpretation concern what constitutes strange behavior, whether or not a 
child continuously repeats things, and whether a child sees imaginary things. 
It appears that low SES parents many over-endorse these items. Parents are 
more often incorrect in their interpretation of critical items for thought 
problems. Caution must be taken when interpreting parent questionnaires from 
a sample of low SES parents, where the questions addressed may not be 
understood as intended. This consequence can lead to over- identification of 
serious psychopathology in these children. (JDM) 
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K. Longeway, Ph.D., S Johnson, Ph.D., M. Garwood, Ph.D. and L. Davis, Psy.D. 

Sinai Samaritan Medical Center 
Aurora Health Care, Milwaukee, WI 

Accurate psychological assessment of low SES clients using standardized behavior rating scales 
presents a unique challenge because the interpretation of the items by parents appears to vary from 
that intended in the normative sample. 

Raadal et al. (1994) assessed behavior problems in children from low income families using 
the Achenbach’s Child Behavior Checklist (CBCL) and found that even in a non-clinical, school- 
based sample children displayed a two times higher rate of attention and aggressive problems than 
children in a non-impoverished sample. Thought problems were reported at three times the rate of 
the normal sample, which was not believable in terms of the base rate of psychosis in childhood. 
Raadal speculated that either the CBCL was invalid for that predominantly low income sample, or 
that child psychopathology may actually be higher in groups stressed by poverty. It appears that the 
CBCL may give false positives for thought problems in this population. This would be a potentially 
serious error, given the severity of diagnoses, e.g. psychosis and pervasive developmental disorder 
(PDD), associated with elevations on this scale. 

The authors wish to thank the SSMC Child Treatment Team, Larry Marx M.D. (Aurora Health Care), and Jerry 
Jacobson M.A. (Department of Medicine, University of Wisconsin Medical School) for their assistance with this project. 
This paper was presented as a poster at the 108 th Annual APA Convention in Washington D.C., August 2000. 
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In our study we wished to explore the validity of the CBCL Thought Problems sub-scale for 
an urban low income population. We hypothesized that cultural, linguistic or SES factors may 
influence the manner in which parents understand and respond to items on the Thought Problems 
scale. 

Preliminary analysis of the CBCL data for 46 children referred to a hospital based mental 
health clinic indicates that more than a third of the sample was rated in the clinical range for thought 
problems in contrast to the less than 5% of the same sample diagnosed with psychosis or PDD by 
a clinic psychiatrist or therapist. 

The CBCL items that appear sensitive to misinterpretation are “strange ideas,” “strange 
behavior,” “repeats acts over and over,” and “sees things that aren’t there.” It appears that low SES 
parents may over-endorse these items. The manner in which these items may be misunderstood was 
analyzed. Individual write-in responses by both parents and teachers were categorized into 
“appropriate” interpretation of the question vs. “inappropriate” interpretation of the item. For 
example, a correct interpretation of “repeats acts over and over,” intended to elicit unusual 
compulsive behavior, would be “drew the same picture of a school bus for weeks.” An incorrect 
interpretation of the same item would be “snacks, sneaking dessert,” a relatively common child 
behavior. Parents were more often incorrect in their interpretation of critical items for thought 
problems than were teachers. It is concluded that parent questionnaire responses must be interpreted 
with caution in urban low SES samples where questions may not be understood as intended, leading 
to the over-identification of serious psychopathology in these children. 
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Introduction 



Child clinicians in urban mental health clinics are frequently asked to assess the possibility 
of emergent thought disorder or psychosis in children who stand out as “different” from children 
with the more common disruptive behavior disorder presentations. There are some differences 
emerging in apparent frequencies of child psychopathology among lower socioeconomic groups 
which may reflect real differences in child disturbance or may be, in part, an artifact of the 
instruments used to assess these children. Kazdin (1992) has reported that lower parent income and 
educational level is associated with higher rates of dysfunction among their children. Raadal et al. 
(1994) , in an article assessing behavior problems in 5 to 1 1 year old children from low income 
families with the widely used Child Behavior Checklist (CBCL), found that even in a non-clinical, 
school-based sample, children displayed two times higher rates of attention and aggressive problems 
than children in the standardization sample. Although the authors did raise the possibility that the 
CBCL may not be valid for predominantly low income parents, they also noted that child 
psychopathology may actually be higher in groups stressed by poverty. Several previous studies 
(Evans, 1975; McIntyre and Keesler, 1986) support the latter hypothesis, particularly for 
externalizing disorders. Yet Raadal’ s finding that thought problems were reported at three times the 
rate of the normative sample was not believable in terms of the expected base rate of psychosis (less 
than 1 % for an adult population and far less than this before young adulthood). Even when the 
pervasive developmental disorders , which are also quite rare with an estimated incidence of 0.5% 
(Simms and Schum, 2000), are combined with psychotic disorders, the overall incidence of thought 
disorder seen in child clinic samples should be very low. 
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Spuriously high elevations on the Thought Problems scale would be a particularly serious 
error in view of the severity of clinical diagnoses associated with such scores. 

The CBCL (Achenbach,1991) is one of the best studied and widely used paper-and pencil 
checklists for parents with established reliability in clinical and nonclinical populations. Yet it has 
been suggested (Raadal et al 1994, Latkovich, 1996) that the normative sample on which the CBCL 
was developed (over 80% Caucasian ) does not match the population of urban African American 
children with whom it is frequently used. There is concern about the validity, or extent to which the 
checklist measures what it is intended to measure in this group. It is also possible that some items 
might be more frequently misunderstood by this group, i.e. content bias, than by the parents in the 
normative sample, decreasing the usefulness of the CBCL in this group. Cultural, linguistic, and/or 
socioeconomic factors may influence how parents understand and respond to items on the Thought 
Problems scale, causing an elevation in scores which is not reflected in concurrent methods of child 
evaluation, e.g. therapist’s or psychiatrist’s interview assessments. 

It seemed that the CBCL might be yielding false positives for thought problems in our 
population. We were interested in determining, in an urban, low SES population, whether 
endorsement of thought problem items corresponded to the psychiatrist’s or therapist’s diagnosis of 
the child. We were also interested in looking at the specific ways in which thought problem items 
might be misunderstood in this sample. 



Method 



Forty-six children seen at an urban hospital outpatient mental health clinic served as subjects. 



Males outnumbered females by approximately 3:1. There were 35 males and 1 1 females. Thirty-two 
of the children were African -American (69%) and thirteen Caucasian ( 3 of these children were 
listed by their parents as Hispanic, and the racial background of one child was unlisted). The ages 
of the children ranged from 4-17, clustering predominantly in the 7-9 year old range. The modal age 
was 8. A majority of the children were on public assistance (Title 19 or medicaid assigned to 
HMO’s). The demographics of the patient population have remained consistent with a previous 
study of therapy outcome at the same hospital nearly a decade ago (Longeway, K. and Glicklich, 
L., 1991). 

All of the children participated in an intake evaluation with a clinic therapist (either a Master’s 
level clinician or a Clinical Psychologist). A provisional diagnosis based on at least two assessment 
sessions with the child and parent was assigned. Assessment sessions generally consisted of a 
background interview with the parent, and a diagnostic/play interview with the child. The 
Achenbach CBCL was generally given to the parent to complete independently, and was scored 
using the computerized package by a psychology intern. The majority of CBCL forms were 
completed by the child’s mother. Forms were also sent to each child’s teacher. Provisional 
diagnoses were determined by the therapists independent of CBCL results. 

Twenty-one of the children were referred to a Board Certified Child Psychiatrist for 
consultation and/or psychopharmacological evaluation. The psychiatrist generally had access to the 
therapist’s assessment results, and in some instances, to the results of the CBCL. The checklist 
produces clinical scales in the form of T-scores which have a mean of 50 and a standard deviation 
of 10. A T-score of 70 on the Thought Problems scales was considered deviant from the norm since 
it was two standard deviations above the mean. 

Analysis of the items which comprise the Thought Problems scale was undertaken as well. 
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Specifically, the items which allowed respondents to write-in a description of the child’s behavior 

were extracted to determine whether parents and teachers made correct interpretation of the questions 

posed; that is, whether they understood the clinical intent of the question. These items were: 

9. “can’t get the mind off certain thoughts” 

40. “hears sounds and voices that aren’t there” 

66. “repeats acts over and over-compulsions” 

70. “sees things that aren’t there” 

84. “strange behavior” 

85. “strange ideas” 

Two doctoral level therapists independently sorted all of the parent and teacher write-in 
responses into two categories — “appropriate” vs. “inappropriate” interpretation of the items. The 
raters were blind to the child’s diagnosis during the sort. 



Results 



As expected, in view of the low incidence of psychosis and PDD in children, only three (7%) 
of the 46 subjects were provisionally diagnosed with thought disorder or PDD by a clinical therapist, 
and only one was ultimately diagnosed with PDD by the child psychiatrist . The actual diagnoses 
given these children by clinicians are shown in table 1. 

However, approximately 46% of children who had completed data sets (i.e. a parent- 
completed CBCL plus diagnosis by a therapist and psychiatrist) scored in the clinical range for 
thought problems. 

We were interested in how the critical items that contributed to the thought problem 
elevations might be understood by parents. Two child clinicians sorted all of the parent and teacher 
write-ins into two categories: “correct interpretation of the question” and “incorrect interpretation 
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of the question.” A statistically significant Cohen’s kappa (kappa=0.70, p<.001) indicated that the 
inter-rater reliability was better than chance. Eighty-five per cent of the write-ins were classified 
similarly by the raters. 

There were more inappropriate write-ins than appropriate ones (40 incorrect, 26 correct). 
Parents, as opposed to teachers, were responsible for the majority of incorrect write-ins. Sixty-three 
per cent of the inappropriate write-ins were by parents (See figures 1 and 2.) 

It appeared that certain questions (66-“repeats acts over and over,” and 84-“strange 
behavior”) were the most often misunderstood by parents. Examples of inappropriate write-ins to 
item 66 were: “kicking like power rangers,” “stealing” and “snacker, sneaking desserts.” 
Inappropriate write-ins to item 84 were “will use bad language out of the blue, has done flips in the 
hall” and “harming others, thinks it’s funny.” These comments by parents did not seem to be 
describing bizarre, repetitive, or ritualistic behaviors of the type associated with true thought 
disorder, but rather common child misbehavior. 

Discussion 

Our retrospective inspection of the CBCL Thought Problem scores of urban, low SES 
children does support the impression that the CBCL overidentifies thought problems in this 
population. In our clinic sample there were no “false negatives”, i.e. failures to identify disordered 
thinking. However, a large number of “false positives” were noted. These children received other 
diagnoses, most commonly attention deficit disorder (ADHD), disruptive behavior disorder, 
dysthymia and adjustment disorder. 
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There are at least three possible explanations for the finding of excessive thought problem 
elevations reported here. First, it is possible that cases that were puzzling to clinicians were given 
CBCL’s disproportionally. That is, children who presented as “odd” or having unusual thought 
patterns might have been more frequently given the CBCL’s for clarification of diagnosis. Yet even 
if the cases were not entirely representative of the clinic as a whole, and included more children with 
possible disordered thinking, neither their intake diagnosis (made before the CBCL results were 
available) nor their discharge diagnosis reflected this. 

Secondly, a “reverse social desirability” phenomena could be occurring. Economically 
disadvantaged parents may feel that their concerns will not be taken seriously unless the severity of 
symptoms is extreme. They may endorse items more readily in order to ensure that their child will 
receive treatment services. This could partially account for the divergent outcomes of rating scale 
scores and clinical diagnoses. Since the latter included an interview with the child, parent reports of 
behavior problems were tempered by direct observation. We must also consider whether there is a 
difference in threshold rates for childhood behavior problems among subgroups of parents. That is, 
what is considered “strange” for low SES parents may include a host of behaviors that clinicians 
from other SES backgrounds consider age appropriate. 

A third conclusion could be drawn from the findings related to write-in comments. The 
majority of these comments were categorized as “incorrect interpretations” by two independent 
raters. Thus, the high proportion of children who scored in the elevated range for Thought Problems 
in this sample may truly reflect differences in how items are interpreted by urban, low SES parents 
as compared to parents included in the standardization sample of the CBCL. 

In many cases the write-in comments described disruptive, defiant and/or hyperactive 
behaviors. Perhaps adding additional modifiers to some items on the Thought Problems scale would 
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indicate that a finer distinction is being sought. For example, item 84 on the CBCL is “strange 
behavior.” If additional adjectives such as “bizarre,” “weird,” or “unusual” were included perhaps 
more accurate responses would be secured from this population. 

Oesterheld and Haber (1997) used a focus group format to assess cultural and linguistic 
patterns of interpreting the CBCL and Conners Parent Rating Scale by native American mothers. 
The authors found that Dakotan/Lakotan mothers had difficulty responding to ten CBCL items. 
Mothers reported that they were unable to comprehend certain words or idioms on the scale, and that 
phrases on the scale implied dominant culture values. Mothers also reported beliefs that their 
responses could be misunderstood by members of the dominant culture. Oesterheld and Haber’s 
findings seemed to support the idea that while items may have both face and construct validity for 
parents included in the standardization sample of the CBCL, there may be subgroups of parents for 
whom many items will be confusing and thus elicit inappropriate answers. 

The potential costs associated with the differing interpretation of items by certain subgroups 
of parents on the Thought Problems scale of the CBCL are high. Mislabeling of children certainly 
can lead to overmedication and more restrictive settings for care. The disproportionate number of 
elevations on this scale reported by Raadal et al. (1994) and now found in the present study may 
indicate that the Thought Problems scale is invalid for low SES groups. 

There are significant methodological improvements which would be required to report 
conclusively on the use of the CBCL in low SES populations. Larger sample size, uniform clinical 
diagnostic procedures, and random sampling of clinic-referred children would certainly increase the 

a 

rigor of the findings. There seems to be mounting evidence, however, that objective parent report 
measures such as the CBCL may be more vulnerable to the influences of class, culture and linguistic 



style than previously considered. 



While the CBCL is still a very useful broadband behavioral inventory, it is important in low 
SES samples to query responses to items on the Thought Problems scale prior to scoring. In addition, 
profile results should be used as only one component of a more comprehensive evaluation. 
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Table 1: Clinicians’ Diagnoses of Children with Significant 
CBCL Thought Problems Elevations 



Therapist’s Intake Diagnosis Diagnosis at Discharge MD Diagnosis 


Post traumatic stress disorder 


Disruptive behavior disorder NOS 


Disruptive behavior disorder NOS 
R/O Dysthymia, RIO PTSD 


ADHD 

PDD-NOS 


ADHD 

PDD-NOS 


Disruptive behavior 


Dysthymia 


Adjustment disorder 
ADHD 




ADHD 


ADHD 


ADHD 

Oppositional-Defiant disorder 


Oppositional-Defiant disorder 


PDD-NOS 

Oppositional-Defiant disorder 


Oppositional-Defiant disorder 
R/O Dysthymia, R/O Psychosis 


Disruptive Behavior disorder NOS 
R/O Depressive disorder 


Disruptive Behavior disorder 


— 


Post traumatic stress disorder 


Post traumatic stress disorder 


ADHD 

R/O Post traumatic stress disorder 


ADHD 


ADHD 




ADHD 


ADHD 


— 


Dysthymia 


Dysthymia 


Oppositional-Defiant disorder 


Dysthymia 


Dysthymia 




ADHD 


ADHD 


ADHD 


ADHD 


ADHD 


PDD-NOS 

ADHD 


Adjustment disorder with anxiety 


Adjustment disorder with anxiety 




Post traumatic stress disorder 


Post traumatic stress disorder 
Conduct disorder 




Adjustment disorder with anxiety 
ADHD 


Adjustment disorder with anxiety 





PDD-NOS = Pervasive Developmental Disorder, not otherwise specified. 
ADHD = Attention Deficit Hyperactivity Disorder 
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Figure 1 




Total Number of Write-Ins to 
Thought Problems CBCL Items 



Figure 2 




Percentage of Inappropriate Write-Ins 
Contributed by Parents and Teachers 
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